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Abstract 

An evaluation of application of the conceptual mapping technique (Inglis, 
2003) to the team-based design of fourteen courses in an Engineering 
undergraduate program is described. The evaluation employed a method of 
scoring each conceptual map against criteria tied to the objectives of the 
design process. The evaluation method was found to be capable of revealing 
differences in application of the conceptual mapping technique. The 
evaluation indicated that the course teams concerned appeared to have 
focused closely on the adequacy of intended learning outcomes blit less 
closely on matching the student assessment adequately with the intended 
learning outcomes. The evaluation also indicated that to obtain the full 
benefit of use of the technique, more training of instructional design staff was 
required than had been provided. 
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Introduction 

Course renewal in Engineering at RMIT University has received major attention over the past five 
years with significant funding and other resources devoted to the task. (In this paper, the meaning 
of the term ‘course’ refers to a modular element of academic study and is synonymous with the 
words ‘subject’ or ‘unit’, sometimes used in the Australian context). Five years ago the Faculty of 
Engineering established a team of instructional designers to provide individual support to 
academic teaching staff. Within each budget year, a strategic analysis of priorities targeted a set of 
courses for renewal. Academic and teaching staff with responsibilities for coordination of a 
selected course were granted time release to undertake the renewal project and an instructional 
designer was assigned to consult with the academic/ teaching staff in a team relationship. 

Course renewal was implemented as a two-phase process. The first phase involved the learning 
design process. The term ‘design’, in this context, does not refer to layout, typography and use of 
illustrations, but rather to the establishment of learning outcomes and associated performance 
criteria, setting assessment measures for each outcome, selecting and defining necessary resources 
and specifying the nature of the learning activities associated with each outcome. The learning 
design phase also involved estimation of the time required for completion of each activity and for 
completion of the set of activities as a whole, so as to ensure that student workloads would not be 
excessive. This first phase of course renewal led to the development of what is termed a 
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‘conceptual map’ (Inglis, 2003; Inglis and Armstrong, 1993). A conceptual map is a document that 
specifies the design of a course. It identifies the key design elements and the relationships amongst 
them. A concise explanation of the process involved in development of a conceptual map is given 
below while a detailed explanation appears in Inglis (2003). The second phase of course renewal, 
involved development of a comprehensive learning package, including a learning guide developed 
in accordance with predetermined standards. 


Objectives of the study 

The purpose of this investigation was to establish the extent to which implementation of the 
conceptual mapping process had resulted in an improvement in the quality of information 
generated by the course development team during the design phase. In other words, the 
investigation attempted to gauge the extent to which the conceptual mapping technique achieved 
the first of its intended purposes discussed above. More specifically, the study was undertaken in 
order to ascertain whether or not introduction of the conceptual mapping process had resulted in a 
discernible shift in the specificity with which Engineering Faculty teaching staff specified key 
design attributes. The study made no attempt to evaluate the adequacy of the coverage of the 
courses. This is a matter that falls within the province of the academic/teaching staff responsible 
for the course, and in turn the academic/teaching team responsible for the program as a whole. 


The process of courseware design 

The literature of instructional design (ID) contains many prescriptions for documenting 
courseware designs. Some design methods are concerned with the organisation of subject matter. 
For example, Reigeluth, Merrill and Spiller (1980) described the Elaboration Theory for selecting, 
sequencing and synthesizing subject matter. At a more basic level, Novak and Gowan (1984) 
described a method for representing knowledge structures and Fambiotte, et al. (1989) described 
and compared a wide range of knowledge mapping approaches and summarised the research 
regarding the actual and potential uses of knowledge maps in education. Jonassen (1993) also 
examined the use of mapping techniques for representing and conveying knowledge in a teaching 
context. Other methods have been described for representing a wider range of learning types than 
just the acquisition of knowledge. It is rare however to see any of these methods being used in 
practice at the tertiary level. One reason for this situation is that most tertiary teachers are not 
familiar with the ID literature. Another important reason is that few tertiary teachers have the 
opportunity to work with experienced instructional designers. Academic teaching staff who have 
had no training in education can sometimes think of the teaching process in terms of transmitting 
knowledge. This tendency has led to teaching that is characterised by transmission of subject 
matter and reflects a ‘transmission theory of learning'. Although, given that what is being referred 
to is a form of teaching practice, it would more appropriately be described as reflecting a 
‘transmission model of teaching’. Moving beyond a transmissive approach requires a teacher to 
think more deeply about a broader rage of learning outcomes and to conceptualise learning as 
involving more than simply the acquisition of knowledge. 

Distance education providers commonly utilise a course team for the development of courseware. 
In a course team context, the member, or members of the team responsible for the curriculum are 
often referred to as subject matter experts (SMEs) (Keppell, 2000). SME’s obviously include the 
appropriate academic/teaching staff members, but may also include external experts brought in to 
contribute to the development of a course. A course team would normally comprise at minimum a 
SME and an instructional designer, however it may include two or more SMEs, an editor and 
media production staff. The course team model enables a range of expertise to be brought to bear 
on the design and production of courseware. Input from an instructional designer can often result 
in a more holistic learning design and development process and avoid a sole focus on content 
which sometimes characterises the approach of SME’s working in isolation on course preparation. 
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The development of courseware, and particularly the development of interactive multimedia 
courseware, involves a substantial investment of human resources. This investment needs to be 
based on a sound learning design methodology if it is to return learning outcomes satisfying a pre¬ 
determined performance specification. Part or even the whole of this investment may be wasted if 
the basic assumptions on which the courseware is structured are flawed. The main purpose of 
developing a conceptual map is therefore to provide a separation between the design and 
development phases of courseware preparation and to document specific outcomes of the design 
phase. In this context ‘design’ refers to decisions taken in relation to the ‘what’ and ‘how’ 
associated with student learning and the resources that are to be provided to facilitate this learning. 
‘Development’ refers to preparation of the materials themselves: the writing of textual materials by 
the subject matter expert(s), and the assembly of other resources. Given that analytic thinking and 
systematic problem solving are so fundamental to the practice of engineering, and as a foundation 
for design and project execution, it has not been an overly difficult task to work with engineering 
academics to translate these principles to the tasks of conceptual mapping and educational design. 


The conceptual map as a courseware development process, delivering a specification of outcomes 
and an educational design methodology, is analogous to the formal requirements analysis, 
specification of outcomes and design proposal phases of an engineering design project. The 
conceptual mapping method has been developed over some years across a range of disciplines 
including careers education (Inglis and Armstrong, 1993), art and design (Inglis and Bradbeer, 
1996), and nursing. However, use of the method in the discipline of engineering represents its 
most extensive application to date. A conceptual map is meant to serve three distinct purposes: 

(1) to specify and document the design of a course to a level of detail that is sufficient to support 
the subsequent development phase; (2) to provide a shared context for discussion between the 
subject matter expert and the instructional designer in relation to course design; and (3) to provide 
a starting point for consideration in a subsequent revision cycle. A conceptual map provides a 
vehicle for steering courseware development and as such, it describes the functional aspects of the 
educational product rather than the characteristics and performance capability of a physical 
product, as may well be the practice in many engineering design approaches. 


One of the attributes of a well-designed conceptual map is its parsimony. The conceptual map is 
meant to serve as a means of communication between the instructional designer and the SME(s). 

It is a working document and is not meant to reach the student. It is not meant to include learning 
material or resources. It is therefore important that it not be filled with so much unnecessary detail 
that the overall structure of the course is lost. The information provided in the conceptual map 
should be succinct. On the other hand, it is important that a conceptual map provides sufficient 
information to support the subsequent development stage. If insufficient information is provided at 
the design stage then time will be lost during the development phase while the missing details are 
sought. Part of the skill of producing a well-designed conceptual map therefore lies in striking an 
appropriate balance between specificity and succinctness. When developing a conceptual map, 
teaching staff were advised to be as parsimonious as possible. 

The structure of the conceptual map is intended to encourage teaching staff to move in particular 
directions while engaging in the design or redesign of a course. It is intended to have staff focus on 
the most important elements of a course’s design: the intended learning outcomes, the learning 
activities through engagement in which the student will be expected to attain the intended learning 
outcomes, the learning resources needed to support the activities, and the types of assessment that 
will enable students’ attainment of the intended learning outcomes to be confirmed. However, it is 
also intended to focus the teaching staff member’s attention on the interrelationships between 
corresponding elements, especially between corresponding learning outcomes and learning 
activities, between corresponding learning activities and learning resources, and between 
corresponding learning outcomes and assessment methods. In this respect the method is consistent 
with Biggs’ (1999) constructive alignment model, although it predates Biggs’ model and goes 
beyond it insofar as it includes other pedagogical elements and is focussed on courseware design 
rather than curriculum design. Finally, consideration of the time dimension, which normally takes 
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place towards the end of the design phase, is intended to focus attention on the magnitude of the 
workload implied by the learning activities that have been chosen so that this can be kept within 
the limit set down for the particular course. For an example, see Inglis (2003). 


Method of investigation 

The method used in the study was to score each of a number of conceptual maps developed for a 
range of specific courses against a set of criteria tied to the objectives of the design process. The 
conceptual mapping process focuses on certain aspects of the design of courses (i.e. the 
identification of particular design elements and the specification of the interrelationships amongst 
them). The study therefore used measures that corresponded to the type of design output the 
conceptual mapping process seeks to generate. 

Design of the scoring instrument 

The method used to score conceptual maps did not employ the conventional type of Likert rating 
scale. Nor did it require a judgement to be made of the quality of the structure and content of 
conceptual maps. Rather, it sought the identification in each case of the number of individual 
elements that met or exceeded a stipulated criterion. It was considered likely that this type of 
judgement would be made more reliably than allocating a score on a rating scale. The measures 
used in each case were ratios (estimated as percentages). Responses were recorded using an 
interval scale with intervals: 0-25, 25-50, 51-75 and 76-100, recoded 1, 2, 3, and 4, respectively. 
These intervals were chosen as being sufficiently narrow as to permit differentiation between 
conceptual maps that are well documented and those that are not so well documented, yet not so 
narrow as to make selection of the most appropriate response category a difficult choice. 

The criteria for each of the measures were chosen so as to reflect the goals of the instructional 
design staff in working with SMEs. The conceptual maps evaluated in the study describe the 
learning outcomes, the learning activities, the learning resources, the time that it is expected 
students will require to complete the activities and the types of assessments that will be used to 
measure students’ attainment of the learning outcomes. However, the adequacy of a conceptual 
map may be further measured in terms of the quality of the information provided and its 
relationship to other items in the conceptual map. 

Definitions of the measures 

The reliability of the scoring method depended on gaining a high degree of agreement on 
interpretation of the response scales. Each of the measures was therefore defined explicitly. The 
definitions are shown in Table 1. A more detailed explanation of each of the measures follows. 
The label for each measure identifies the relevant component of the conceptual map and the 
quality being assessed. 


Table 1. Measures used in scoring conceptual maps 


Dimension 

Measure 

Learning outcome 

1 . 

Appropriateness 

Learning outcomes corresponding to capabilities required of a 
professional engineer 

2. 

Level 

Learning outcomes that clearly require more than rote memorisation 

3. 

Generic capabilities 

Relevant generic capabilities that are represented in the list of intended 
learning outcomes 

4. 

Assessability 

Learning outcomes specified such that they are capable of being directly 
assessed 

5. 

Elaboration 

Terminal outcomes that have been elaborated to a depth sufficient to 
enable the entry capabilities of at leats 75% of learners expected to be 
taking the course to be matched 
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Learning activity 

6. Match to learning 
outcomes 

Activities that provide a satisfactory match to intended learning 
outcomes 

Time 

7. Estimate 
plausibility 

Activities for which the resources listed appear sufficient to complete 
the specified activities 

Resources 

8. Sufficiency 

Activities for which the resources listed appear sufficient to complete 
the specified activities 

9. Specificity 

Resources identified in sufficient detail to allow compilation of the 
learning package without further research 

Assessment 

10. Coverage 

Learning outcomes for which the proposed assessment appears to 
provide an adequate measure 


Learning outcomes — Appropriateness 

Often students are expected to demonstrate capabilities that have little or no relevance to their 
future professional roles. For example, students may be required to provide a definition of a 
particular concept (which they have already been provided and therefore can memorise) whereas 
the capability they would require in a professional context would be to recognise examples of the 
concept. When students are expected to demonstrate capabilities that would never be needed in a 
professional role, this is usually because the designated capabilities are either more easily assessed 
or more easily taught in a formal educational setting. This measure gives the proportion of those 
learning outcomes that are listed that describe capabilities that students may need to employ in 
their future professional roles. 

Learning outcomes — Level 

Learning outcomes can be categorised into different types that vary in terms of the level of 
learning implied (Gagne, 1985). Rote memorisation is considered to involve low level learning 
whereas problem solving is considered to involve high level learning with concept learning falling 
somewhere between. Rote memorisation is generally considered an inappropriate type of learning 
outcome in tertiary education except in relation to certain areas where the ability to recall specific 
information is important. The term ‘rote memorisation’ refers to the situation where the focus is in 
the words themselves rather than the meaning behind the words (Mayer, 2002). Yet it is 
recognised that learning outcomes are commonly written in such a way as to expect rote 
memorisation. More desirable are outcomes that expect students to engage in formation of 
concepts, learning of principles or problem solving. Whether a particular learning outcome expects 
rote memorisation depends on the student’s previous learning. Merrill (1971) pointed out that the 
nature of learning is such that learners are constantly striving to push their learning down to a 
lower level. Consequently, what is concept learning for one learner may be memorisation for 
another. The cognitive level at which learning outcomes are pitched therefore depends on taking 
into account the learner’s existing capabilities. In a class situation this implies that the design of 
learning materials allow for the student whose capabilities in the domain in question are least well 
developed. 

Learning outcomes — Generic capabilities 

As indicated. Engineers Australia promotes a set of generic attributes that graduates in engineering 
should be capable of demonstrating. This measure compares the number of generic capabilities 
that it is judged could be reasonably reflected in the learning outcomes for the course with the 
number of generic capabilities that are actually reflected in the learning outcomes for the course. 
Using this measure requires an assessment to be made first of the generic attributes that ought to 
be reflected in the learning outcomes for the course. Once a list of the course-specific generic 
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attributes had been prepared, the number of generic attributes actually reflected in the intended 
learning outcomes was compared to the number of generic attributes that could be appropriately 
reflected in the intended learning outcomes for the course. 


Activities — Match to learning outcomes 


The purpose of learning activities is to provide learners with the opportunity to acquire the 
capabilities described in the specified learning outcomes. For learning activities to serve their 
intended purpose they need to be closely matched to the corresponding learning outcomes. If they 
are not closely matched to the learning outcomes students may devote a substantial amount of time 
to a course but still not acquire the capabilities that are expected. This measure compares the 
number of activities judged to be satisfactorily matched to their corresponding learning outcomes 
with the total number of activities that have been set down for completion. 

Time — Estimate plausibility 


Academic teaching staff are sometimes more able to make a much more accurate estimate of how 
much time a student is likely to take on an individual activity than how much time a student will 
take to complete the full set of learning activities for a course. The method of preparing the 
conceptual map therefore requires SMEs to record their estimate of the time that they expect 
students will typically require to complete each individual activity. In practice, once all activities 
have been specified, the time requirements are summed and if the sum varies by more than 10% 
from the total learning time nominated for the course, the staff member is encouraged to consider 
modifying the requirements of the course so as to adjust the workload. This measure compares the 
number of activities for which the time estimate seems plausible with the total number of 
activities. A low ratio may either indicate that no estimate has been given for the time required for 
many activities or that the estimates that have been given are not plausible. 


Resources — Sufficiency 


When students are learning from a learning package or online, their ability to complete the 
activities specified for a course depends on whether all the resources required for the activities 
have been provided. In some cases the resources that need to be provided are directions for 
locating materials from elsewhere rather than the materials themselves. To ensure that all the 
required resources are provided, the SME needs to work systematically through each activity, 
identifying what is needed, and gathering those materials and transforming them into suitable 
forms. Each of the resources required should be identified in the conceptual map. This measure 
compares the learning resources that are identified in the course design process with the learning 
resources that, in the judgment of the researcher, would be needed for completion of the activities. 


Resources — Specificity 


The purpose of identifying the resources required for completion of the activities is to enable the 
resources to be assembled in the subsequent development stage. Each item need only be specified 
in sufficient detail that it is clear to all of the members of the course team as to what is meant. For 
example, it would be quite sufficient to identify an extract from a book by the author, year of 
publication and page number(s) if the members of the team knew exactly which book was being 
identified from the author and year. Later, when a reference to the book is included in the learning 
package the full bibliographic details would be cited. This measure indicates the proportion of the 
resources that are listed that are specified in sufficient detail to allow the learning package to be 
compiled without further research. 


Assessment — Coverage 


Within the Engineering Faculty an attempt has been made to increase the comprehensiveness of 
assessment in courses. The general principle that is being advocated is that if attainment of a 
learning outcome is important, then attainment of the learning outcome should be assessed. The 
conceptual mapping technique contributes to the achievement of this goal by having course teams 
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identify the method by which it is proposed the attainment of each learning outcome will be 
assessed (e.g. by a problem in the examination, by a practical report, by one of more object test 
items). This measure indicates the proportion of the intended learning outcomes for which the 
proposed assessment is judged to be adequate. 


Scoring method 


The conceptual maps were scored by both researchers independently. The researchers then came 
together to compare their scores. Where a discrepancy between the scores on an individual item 
was found, the basis on which the scores had been assigned were discussed and consensus was 
reached on an appropriate score for that item. By this means an agreed score sheet was produced 
for each conceptual map. 


Such documentation as previously existed for each course was then scored. The documentation 
that existed prior to establishment of the conceptual mapping process was much less substantial 
than that provided in the conceptual map. In most cases the only documentation that was 
previously available was the Course Guide provided to students when they enrolled in a course. 


What was being looked for in this study was evidence of improvement in the quality of the 
information generated at the design stage. An inherent assumption of the conceptual mapping 
process is that the first attempt at generating a conceptual map will not produce the most ideal 
result. Each iteration of the design-development-production-delivery cycle provides an opportunity 
for improving the design of a course further. 

Selection of conceptual maps for scoring 

Each member of the team of instructional designers was asked to select two conceptual maps from 
amongst those in which they had worked. It was indicated that the conceptual maps should be 
chosen as being representative of those on which each instructional designer had worked and 
which they considered were developed to the point that they were ready to be used in the 
subsequent development stage. The first author of this paper, as a member of the instructional 
design team and originator of the conceptual mapping technique also selected two conceptual 
maps. All conceptual maps were then scored. 


Some weeks later, after allowing for further development of remaining projects, the members of 
the instructional design team were asked to select two further conceptual maps. These were also 
scored. 


Results 

In a study of this type one would normally wish to undertake a before-and-after comparison of the 
impact of the change in practice on the quality of what is being produced. However, in this case 
not only did the introduction of the conceptual mapping technique mark the first attempt to focus 
attention specifically on the design of courses as an activity separate from the development process 
but it also coincided with initial development of learning packages for distance education. There 
was no previous equivalent practice with which the new approach could validly be compared. 

The aim of this study was therefore not primarily to establish whether adoption of development of 
conceptual maps had led to improvement in documentation of the design decisions, but to establish 
the effectiveness with which the conceptual mapping technique was being applied. A secondary 
aim was to determine the suitability of the scoring method as a means of gauging the adherence to 
the design principles embedded in the conceptual mapping technique. 

The scores given to each of the conceptual maps on each of the ten measures is shown in Table 2. 
Because of the relatively small number of cases and the small number of scoring categories no 
attempt was made to analyse the data for variability. It was considered that this would convey a 


50 







DEE 


Journal of Learning Design 


Inglis and Bradley 


misleading impression of the precision of the data and that the variability of the data was therefore 
best judged by inspection. 


Courses 1 and 2 in the table were the courses to which the first author contributed as instructional 
designer. The remaining twelve courses were those to which the other members of the instructional 
design team contributed. Courses 3-9 were the remaining courses in the first batch to be scored. 
Courses 10-14 formed the second batch. 


Table 2. Scores on the ten measures for each conceptual map 


Measure 

Course 

Mean 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

11 

12 

13 

14 

Appropriateness 

4 

4 

2 

2 

4 

3 

3 

3 

4 

4 

3 

4 

4 

3 

3.36 

Level 

4 

4 

2 

3 

2 

3 

3 

3 

3 

3 

4 

4 

4 

3 

3.4 

Generic 

Capabilities 

2 

3 

1 

2 

1 

2 

1 

3 

1 

1 

3 

1 

1 

1 

1.64 

Assessability 

4 

4 

4 

4 

4 

4 

4 

4 

4 

4 

4 

4 

4 

4 

4 

Elaboration 

4 

4 

2 

3 

2 

4 

3 

3 

4 

2 

4 

3 

3 

3 

3.14 

Match of LAs to 
LOs 

4 

3 

4 

1 

1 

2 

3 

3 

3 

4 

1 

4 

4 

2 

2.78 

Estimate 

plausibility 

4 

4 

4 

1 

3 

1 

1 

4 

4 

1 

4 

4 

4 

3 

3 

Sufficiency 

4 

3 

3 

2 

4 

1 

2 

4 

3 

1 

4 

4 

4 

3 

3.14 

Specificity 

4 

2 

2 

2 

4 

3 

3 

3 

3 

1 

3 

4 

4 

3 

2.93 

Coverage 

3 

1 

1 

1 

1 

1 

1 

1 

3 

2 

1 

2 

2 

1 

1.5 


Had all design teams applied the conceptual mapping technique to best effect, then each 
conceptual map would have received a score of 4 on each measure. However, the majority of the 
scores were less than 4. From both the individual scores and the average scores, it can be seen that 
the quality of the information fell well below that which was aimed for. Based on the means of the 
scores and using a criterion of 3 or above as acceptable, the design teams were generally able to 
specify intended learning outcomes which described capabilities required of a professional 
engineer, necessitating more than rote memorisation, and were elaborated to an appropriate depth. 

Another aspect of the pattern of scores across the ten measures is the variability of the scores 
across the courses. On most measures scores are reasonably consistent. However, on two measures 
the scores varied widely: the match of learning activities to learning outcomes, and the sufficiency 
of the listed resource materials for completing the learning activities. 


Discussion 

The principle aim of this study was to establish how effectively the conceptual mapping technique 
was supporting the design of courses rather than to evaluate the design efforts of individual staff. It 
was recognised that, because the design approach was new and unfamiliar to the teaching staff, the 
results produced would reflect the inexperience of staff using it. Therefore, of greater interest than 
the overall quality of the results produced were the pattern of the results and the areas where future 
efforts at improving application of the method could best be directed. 

It is not possible to quantify all aspects of a design document. An instrument of the type used in 
this study can therefore only provide a partial indication of the extent to which introduction of the 
conceptual mapping process has altered the way engineering educators think about their courses 
The results of the investigation suggest that course teams in the subjects that were included in this 
study had focussed closely on the learning outcomes that they wanted students to attain. This was 
considered an important achievement because traditionally teaching staff in universities have 
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focussed more on subject matter than on learning outcomes. While the evidence provided by this 
study does not allow one to say conclusively that the use of the conceptual mapping process was 
responsible for a shift in focus, it would seem that it is likely that a shift in focus did in fact occur. 
By contrast, there were two areas where the teams’ efforts appeared to be deficient. The first was 
in encompassing generic capabilities and the second was in providing coverage of the intended 
learning outcomes in the assessment. 


The fact that generic capabilities were not adequately reflected in the overall learning outcomes 
was a disappointing finding, given the importance being placed on the development of generic 
capabilities both by the Faculty and by the engineering profession. However, this is explained by 
the relative lack of experience amongst teaching staff at integrating generic capabilities into the 
engineering curriculum and their uncertainty as to how best to go about doing this. The advice that 
instructional designers were asked to give SMEs was to ensure that generic capabilities were 
reflected in the set of intended learning outcomes wherever this was appropriate. In some other 
universities, generic capabilities are separately assessed (Atlay and Harris, 2000). From discussion 
of importance of generic capabilities in the literature it appears likely that the extent to which 
courses facilitate students’ attainment of generic capabilities is an area of the curriculum that will 
come under increasing scrutiny in the future (Bowden, et al., nd; Nunan, 1999). 


The issue of the coverage of the assessment is a matter on which there is some debate. It is not 
uncommon for teaching staff at university level to assess only a subset of the intended learning 
outcomes that have been identified for a course. Whether or not students’ attainment of the 
intended learning outcomes for a course should be comprehensively assessed is therefore a matter 
on which one is likely to find a variety of opinions. However, there has been some attempt within 
the Faculty of Engineering at RMIT to assess more comprehensively and uniformly than has been 
the case in the past. The very low scores on this measure indicate that this thrust has largely been 
ineffective in the case of the courses included in this study. 


Improvements in the specificity of the design information may arise in either of two ways. The 
form in which the conceptual map documents information calls for a greater amount of 
information to be recorded and for this information to be recorded in a more useful form. 
However, improvements may also result from the greater amount of thought being given to their 
design. Many staff have commented that, as a result of the experience of preparing a conceptual 
map, they have thought much more deeply about the design of their courses. 


Use of a scoring instrument closely matched to the aims of the design process together with the 
independent scoring of each conceptual map followed by the process of reaching consensus on the 
final score provided an objective measure of the quality of the information contained in the 
conceptual maps. What the scoring process revealed was not only a variation in the quality of the 
information generated by the process but also a considerable variation in the application of the 
process itself. 


The ability of course development teams to use the technique is a function of both their 
understanding of the technique and their willingness to persist with the process of documentation 
of the design until that task has been sufficiently advanced. Anecdotal evidence from the 
instructional designers who were involved with the course development projects that provided the 
conceptual maps for this study suggests that while course teams were keen to move on from design 
to development, the variation in the quality of the conceptual maps can best be explained by the 
course teams’ lack of familiarity with the method. 

This finding highlighted the need for the instructional designers to receive more intensive training 
in use of the method. While it had been the standard practice for the members of the instructional 
design team to meet weekly to discuss the progress of projects, these weekly discussions did not 
get down to the level of detail that was capable of revealing the extent of variation in application 
of the technique. This variation is ascribed to the relative familiarity of different instructional 
designers with the attributes of the concepts on which the technique is based, such as learning 
outcomes, resources, and assessment. 
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The purpose of the conceptual mapping technique is to enable course development teams to 
document the designs of courses to be taught with the use of learning packages prior to the 
development of the learning materials. However, the value of the technique depends on how 
effectively course teams are able to apply it in actual course development situations. 

This investigation of the quality of conceptual maps produced by a variety of course teams across 
a range of engineering subjects suggests that while teams had mastered some aspects of the design 
process there were other aspects with which they were having difficulty. 

It is expected that as academic teaching staff become more familiar with the conceptual mapping 
process the quality of the conceptual maps they produce will improve. The maps will become 
more tightly structured and the information provided in the maps will be more specific. However, 
one would expect that these improvements would be less obvious than the improvements in 
moving from earlier forms of documentation to the conceptual map. 

It is anticipated that the conceptual mapping process itself will be further refined over time. 
Changes may be made to the nature and quantity of information recorded in conceptual maps as 
well as the procedures used to produce them. The results of this and future studies will help to 
ensure that any changes that are made contribute to the overall improvement of the process rather 
than simply reflect the personal preferences of those using the process. 
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